feat(agents): reviewer quality package: readable verdicts, calibration, re-review - #12122

…d colors

In spike X, 27 of 29 VQ-01 lines called the style guide's own sizes too
small (labelsize 8, fontsize 8 to 10, "small relative to the 3200 px
canvas"), and the rerun repeats it: 50 of its 56 VQ-01 residual lines are
size complaints about text at the table's values. The reviewer reads the
whole style guide, whose "Common Mistake" note says matplotlib
`fontsize=10` is too small and whose General Rules ask for elements 2-3x
larger than library defaults, while its "Visual Sizing Defaults" table
prescribes exactly those sizes at `figsize=(8, 4.5)`, `dpi=400`. Repairs
then enlarged fonts against the house style, and `ok` stayed out of
reach for correct plots (the owner accepted 22 of 24 needs_attention
renders of spike X).

reviewer.md now:

- judges VQ-01 against the table and quotes it (title 12 pt, axis titles
  10 pt, ticks, legend and annotations 8 pt, about 67, 56 and 44 px at
  dpi 400), says the library-default notes refer to a library's default
  dpi, and forbids reporting a size the table allows;
- judges VQ-07 from the color values in the code, not from a hue's look
  in the render: Imprint colors and theme tokens are compliant, and a
  fit line or outline in its series' color is not a second group (the
  rerun flagged a brand green as "#138A6B" twice);
- lists what is never a defect: the house style and styling the
  catalogue plot already had, the user's own spelling of names, a gate
  note on its own, and an improvement that fails no criterion;
- treats a gate note as a measurement to confirm in the render, not a
  defect to copy, and says `ok` is the expected verdict for a correct
  plot;
- states the 200-character limit of each defect text.

A test ties the quoted sizes to the style guide's table, so a change to
the table fails it until the reviewer row follows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

A rejected first review sent its lines to the repair, and the repaired
render then shipped without a second look (stage `not_rereviewed`), so a
repaired run could never end `ok`. In the rerun of spike X on main, 39 of
244 runs ended that way; the cleaner the gates, the more runs reach a
review on attempt 1 and hit this cap.

The pipeline now allows `MAX_REVIEWS` (two) reviewer calls: the first
review, and one more when a later attempt rendered and passed the host
gates after a review. Its verdict decides the shipped render: `ok` ships
`ok`, a rejection ships `needs_attention` with the second review's lines
(they describe the shipped render), and an unread or skipped (budget,
deadline) second review keeps the first review's lines as today. The
stage vocabulary is unchanged: the second review ends its attempt with
`reviewer_ok`, `reviewer_defects` or `reviewer_unreadable`, and
`not_rereviewed` now means a run that had made both reviewer calls. The
`pipeline_result` line adds `reviews`, the number of reviewer calls, and
`reviewed_attempt` names the last reviewed attempt.

Call bounds stay consistent: root 2 + adapter 2 + reviewer 2 = 6 model
calls per request (7 with the scope judge, which the RunConfig cap does
not count), under `AGENT_MAX_LLM_CALLS` 12. Only the condition of the
`not_rereviewed` branch changed in the attempt loop, so the
`AGENT_MAX_ATTEMPTS` loop of #12119 rebases onto it; with three attempts
the run still makes at most two reviewer calls.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

… budgets

Output caps. The owner's rule (2026-10-10): every cap leaves ample
headroom above the measured answers, because an unused cap costs nothing
and a cut-off wastes the call. In the rerun of spike X on main (244
runs, Claude Haiku 5.5), 23 of 244 first adapter calls ended with
MAX_TOKENS at the 2,048 edit cap, and finished edit plans reached 2,030,
so the cap cut the distribution. The new caps against the largest
measured answer:

- adapter, edits only: 8,192 on both providers (largest 2,030);
- adapter, full file allowed: 16,384 on Claude (largest 2,606; a full
  file at the schema's 24 KB limit is about 9,800 tokens), 12,288 on
  Gemini as before (its soft-deadline arithmetic bounds it);
- reviewer and root: 4,096 (largest 600 and 259);
- judge: 256 (59 in every run).

Fitted on the rerun's run times (R^2 0.95), Haiku 5.5 takes about 0.8 s
per call plus 0.0029 s per output token, so a call that uses the whole
16,384 takes about 49 s, inside the 60 s `ADAPTER_P95_S` reserves, and
a full 4,096-token review about 13 s, inside `REVIEWER_P95_S` (30 s).
The Anthropic SDK accepts 16,384 without streaming (limit about 21,300).
`allow_full_file` now picks the full-file cap by provider
(`ADAPTER_FULL_MAX_OUTPUT_TOKENS`, `GEMINI_ADAPTER_FULL_MAX_OUTPUT_TOKENS`).

Cost-weighted budgets. The request, user-day and global-day token
budgets counted cached tokens at full weight, so a reviewed run with two
adapter attempts booked about 72k of the 80k request budget, a second
review pushed it past, and 1M per user and day held about 14 runs. They
now count input-token equivalents at the list-price ratios
(`ledger.BUDGET_WEIGHTS`: uncached 1.0, cache read 0.1, cache write
1.25, output 5.0; a test ties them to agents/evals/pricing.py, which is
unchanged). The judges book their input and output split the same way.
`RequestLedger.tokens` and `judge_tokens` keep the plain counts for the
`done` event; each `model` attribution line adds the weighted `budget`.

Measured on the same 244 runs (warm prompt cache):

- every run: median 37,157 weighted against 70,488 plain tokens;
- reviewed runs with two adapter attempts (166): median 39,499 against
  72,172, so a run now books about 38k to 40k instead of about 70k;
- a second review adds a median of 11,263, a third attempt 12,145, so
  both fit the 80,000 per request;
- 1,000,000 per user and day holds about 26 median runs (27 at 37,157,
  25 at 39,499) instead of 14.

The settings' numbers stay at 80,000, 1,000,000 and 3,000,000. A cold
prompt cache weighs more: each agent's first call writes its prefix at
1.25 instead of reading it at 0.1. Simulated from the rerun's per-call
counts, a reviewed two-attempt run then books a median of about 77,700
(95th percentile 87,000; 61 of 166 reach 80,000), so on a cold cache the
request budget can stop the second review or replace the root's closing
reply with the budget refusal. The ledger, the settings and the docs say
so; raising the request budget is left to the owner.

Docs: the settings docstring and agents/README.md (a new "Token budgets
and output caps" section) document the caps and the weights; the design
doc's agent table and Budget paragraph follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

…the catalogue

The rerun of spike X on main, with the renderer image that carries the
drawn-text G3 fix, still reported G3 in every run of heatmap-basic
seaborn (12 of 12, 18 gate lines) and scatter-basic seaborn (12 of 12,
22 lines) and in 8 of 12 runs of heatmap-basic matplotlib (8 lines).
Running the probe harness in-process on the three catalogue
originals (ANYPLOT_THEME=light) reproduces each line, and matplotlib's
own tight bounding box and the saved PNGs confirm it:

- heatmap-basic seaborn: the x-axis title "Month" (10 pt) ends 23.2 px
  below the bottom edge of the 2400x2400 canvas; 61 ink pixels sit on
  the last row. The adapted renders report 15 to 16 px.
- heatmap-basic matplotlib: the y-axis title "Department" (10 pt,
  rotated) starts 25.9 px left of the canvas; 154 ink pixels sit on the
  first column. The adapted renders report 56 to 64 px (longer user
  column names).
- scatter-basic seaborn: the title (12 pt, pad 14, top=0.93) rises
  2.8 px above the 3200x1800 canvas; its ascenders put 42 ink pixels on
  the first row. Every adapted render reports the same 3 px.

The renderer bakes DejaVu Sans, the font the local run fell back to, so
the measurements carry over. The probe is right and needs no change;
the clips are input for the catalogue normalisation. The design doc's
spike-X follow-ups paragraph says so in one sentence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

The committed baseline was the first spike-X run of 2026-10-10 (122
runs, report schema 1), taken before the drawn-text G3 fix, so 56 of
its runs shipped with a probe false positive. The rerun of the same
evening replaces it:

- 122 cases, 2 runs each, at most 2 attempts, eu, remote renderer with
  the G3 fix; report schema 2 with stages, attempt logs and per-call
  finish reasons;
- stamp commit 6d0de1e: main at 8de9fb7 plus the AGENT_MAX_ATTEMPTS
  setting of #12119 at its default of 2, which leaves the pipeline as on
  main;
- 221 of 244 runs passed the gates (90.6 %), 57 ended ok (23.4 %),
  $0.99 at list price, median $0.0041 per passed plot.

The file is the report byte for byte (1.08 MB), written with the
harness's own `write_baseline`, so the baseline checks ran (no early
stop, no promoted cases, no run that ended in an error), and
`load_baseline`, `compare` and `summary_markdown` read it with the
current code. It predates the answer-schema counts of this branch, which
the diff does not compare.

agents/README.md, the baselines README, the design doc's status note and
test_baselines.py describe the new baseline.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

changelog.d/agents-reviewer-quality.md covers the branch: the second
review of a repaired render, the output caps, the cost-weighted token
budgets and the new Claude baseline (Changed), and the mended answers
and the reviewer's calibration to the style guide's table (Fixed). The
style-guide one-liner (dcd95f739) has no bullet, because it may move to
its own PR with a review-retest arm.

Docs where they still described a single review:

- docs/concepts/agent-network.md: product step 1, the pipeline sketch
  (`reviews` instead of `reviewer_used`), the `finish` rules, the Bounds
  table (2 reviewer calls, `MAX_REVIEWS`) and the typical call count;
- agents/README.md: the pipeline row, and the report description, which
  now names the `answer_outcomes` and `answer_rules` counts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

… review findings

Request budget. On a cold prompt cache the cost-weighted budget at the
unchanged 80,000 was stricter than main's plain count: each agent's
first call writes its prefix at 1.25, so a reviewed two-attempt run
books a median of 77,747 weighted (simulated from the rerun's calls)
and a second review then pushes the root's closing reply into the
`budget` refusal while the plot ships. The rerun measured the case too:
the first, cold run of area-basic-seaborn-decimal-comma booked 100,282.
AGENT_REQUEST_TOKEN_BUDGET is now 160,000, about 36 % above the
costliest reviewed run simulated cold with a second review (115,046;
about 117,300 with three attempts in the three-attempt rerun). Before
each review the pipeline also keeps REVIEW_RESERVE_TOKENS (35,000: a
cold review at most 28,099 plus the closing reply at most 5,022) of
every token budget and room for two more calls, so a tight budget skips
the review and ships with the first review's lines instead of refusing
the reply to a shipped plot. `budget_allows`, `request_ok` and
`daily_ok` take the reserve. A flow test reproduces the case with a
budget of the reserve plus 600. The settings, ledger, README, harness
docstring and design doc state the measured numbers; `pricing.py` no
longer ties the unmodelled long-prompt tier to the request budget,
which never bounded one call (the largest prompt was 23,809 tokens).

Off-checklist ids. A pass whose only defects name ids outside the
reviewer's checklist (the style guide it reads names VQ-05) is read as
a pass without them; a rejection that names only such ids, or a pass
next to a checklist defect missing its texts, stays unread.
`reviewer.md` says ids outside its table are never used, and the
reviewer docstring and the changelog say precisely what is mended.

Docs. The G3 claim covers the 48 lines from the three examined pairs
and names the 2 lines on adapted renders as unexamined (the drawn-text
probe also measures text that clip_on keeps off the canvas). The
JudgeVerdict docstring says the budgets weigh its split; the harness
section points at the committed rerun baseline and lists the answer
counts; the README marks the cap table's measured column as Claude's
and gives Gemini's plan sizes.

Rebased onto main after #12119 (AGENT_MAX_ATTEMPTS): the loop keeps
`attempt < settings.max_attempts`, and the `not_rereviewed` exit now
waits for `MAX_REVIEWS`; with three attempts, two rejections spend both
reviews and attempt 3 ships unreviewed. The style guide commit
(dcd95f739) is no longer on this branch: it needs its own PR with a
review-retest candidate arm, and is kept on the local branch
fix/style-guide-fontsize-example.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

The per-user and global daily budgets kept the design's 1,000,000 and
3,000,000 after the request budget moved to cost-weighted tokens. With
a warm prompt cache that is about 26 median runs per user, but a cold
reviewed run books about 77,700 weighted tokens, so a user whose
requests keep starting cold reached the daily refusal after about 13
plots, a third of the 40 runs that AGENT_DAILY_PIPELINE_RUNS allows.

AGENT_DAILY_TOKEN_BUDGET is now 2,000,000 (about 54 median warm runs or
26 cold ones, so the run cap binds first on a normal day) and
AGENT_GLOBAL_DAILY_TOKEN_BUDGET 6,000,000, three users at their daily
budget. The settings table and docstrings, the ledger's measured note,
the README, the design doc's budget bullet and environment line, and
the changelog fragment state the new numbers and the reasoning.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

…get wording

Three review findings on #12122.

The judges' output now weighs its own model's list-price ratio
(JUDGE_OUTPUT_WEIGHTS: 5 on Claude Haiku 5.5, 8.33 on Gemini 3.5
Flash-Lite, the agents' 5 for a model outside the table) instead of
the agents' shared 5, so the scope and dataset judges on the Gemini arm
are cost-weighted by list price as the budgets claim. The scope guard
and the dataset route pass the configured judge model; the parity test
checks the table against agents/evals/pricing.py like BUDGET_WEIGHTS.

The reviewer's VQ-01 row no longer turns the style guide's sizing table
into hard minimums: text at or above the table's sizes is never a
defect, text below them is a defect only when it cannot be read in the
render, never for its size alone, because the style guide allows the
deviations a context calls for; the table has no annotation row, and
the prompt says so instead of assigning annotations 8 pt.

The daily budget's docstring, settings table, README, ledger note,
design doc and changelog said the 40-run cap binds first while their
own cold median gives about 26 runs under 2,000,000. They now say both:
about 54 warm runs, where the run cap binds, and about 26 for a user
whose every request starts cold, where the token budget binds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M